{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00bf530",
   "metadata": {},
   "source": [
    "# Project 1: Regression\n",
    "\n",
    "This project asks you to perform various experiments with regression. The dataset we are using is taken from a real estate dataset:\n",
    "\n",
    "https://www.kaggle.com/datasets/mirbektoktogaraev/madrid-real-estate-market\n",
    "\n",
    "The objective of this project is to become familiar with the underlying techniques of machine learning, and implement some of the techniques yourself. \n",
    "\n",
    "You will write code and discussion texts into code and text cells in this notebook. \n",
    "\n",
    "If a block starts with TODO:, this means that you need to write something there. \n",
    "\n",
    "Some code had been written for you to guide the project. Don't change the already written code.\n",
    "\n",
    "## Grading\n",
    "The points add up to 42, that is 30 + 12 bonus points. While there is no difference between the regular and the bonus points, I recommend that you solve the problems labeled \"BONUS\" after you finished the other ones. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a440a191",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import scipy as sp\n",
    "import sklearn as sk"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "691d1a9f",
   "metadata": {},
   "source": [
    "## Setup for the first part of the project\n",
    "\n",
    "For problems P1 to P6 we are using a simple dataset where we extract one \n",
    "explanatory variable ``sq_mt_built`` to predict the price of the house ``buy_price``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "902ccefe",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"houses_Madrid.csv\")\n",
    "print(f\"The lenght {len(df.index)}\")\n",
    "print(f\"The columns of the database {df.columns}\")\n",
    "df[[\"sq_mt_built\", \"buy_price\"]].plot.scatter(x=\"sq_mt_built\", y=\"buy_price\")\n",
    "## FIXME: add here the creation of the training data and test data\n",
    "\n",
    "df_shuffled = df.sample(frac=1) # shuffle the rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54bf2ec2",
   "metadata": {},
   "outputs": [],
   "source": [
    "x = df_shuffled[\"sq_mt_built\"].to_numpy(dtype=np.float64)\n",
    "y = df_shuffled[\"buy_price\"].to_numpy(dtype=np.float64)\n",
    "training_data_x = x[:16000]\n",
    "training_data_y = y[:16000]\n",
    "test_data_x = x[16000:]\n",
    "test_data_y = y[16000:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d8bfd4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_data_y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c63bf75",
   "metadata": {},
   "source": [
    "## P1: Loss function (3 pts)\n",
    "Implement a root-mean-square error (RMSE) loss function between the prediction $\\hat{y}$ and $y$ value using Python operations. Run some experiments to validate that this works as expected. \n",
    "Then, look up the same in the sklearn library\n",
    "https://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics\n",
    "and implement it based on what is there. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66d0d477",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: implement the loss function here using Python math ops and sklearn\n",
    "def loss_RMSE(y, yhat):\n",
    "    return 0\n",
    "\n",
    "def loss_RMSE_sk(y, yhat):\n",
    "    return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a723623d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Now, run some experiments with your function, with the one taken with sklearn \n",
    "# Compare their outputs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0548dea",
   "metadata": {},
   "source": [
    "## P2: Implement a linear predictor (3 pts)\n",
    "Implement a function of type ``predict(x, theta) --> y_hat`` which implements a linear model of the type $\\hat{y} = \\theta_1 \\cdot x + \\theta_0$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b91956b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: implement the predictor function here\n",
    "def predict(x, theta):\n",
    "    y_hat = 0\n",
    "    return y_hat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7af53924",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: now, run some experiments with it"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5fcd1c6",
   "metadata": {},
   "source": [
    "## P3: Implement a \"grid search\" function (3 pts)\n",
    "Implement a function grid_search() which returns an estimate of the best $\\theta$ by trying out all the combinations of possibilities on a grid and returning the values that give you the most values. \n",
    "gridx and gridy define the range of numbers that we want to explore. For instance, grid0 might be [0, 0.25, 0.5, 0.75, 1.0] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9663e31d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: implement the grid search function here \n",
    "def grid_search(training_data_x, training_data_y, grid0, grid1):\n",
    "    theta = [0, 0]\n",
    "    return theta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65e10363",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: run some experiments with grid_search\n",
    "# Define some grid values. Train it on the data set. Test it on the test set. \n",
    "# Print the loss on the data set and the test set. Measure and print how long the training takes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "716ed9d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: repeat the experimentation from above with different grids. \n",
    "# Finally, print the grid that provides the best value while still running faster \n",
    "# than 10 seconds."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c46a9f4",
   "metadata": {},
   "source": [
    "## P4: Implement a random search function (3 pts)\n",
    "Implement a function that returns the estimate for the best $\\theta$ by trying out random \n",
    "$\\theta=[\\theta_0, \\theta_1]$ values, and returning the one that minimizes the error on the training set passed to it. The number of tries is described in the ``trials`` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc373403",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: implement the random search function here \n",
    "def grid_search(training_data_x, training_data_y, trials):\n",
    "    theta = [0, 0]\n",
    "    return theta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "294dde6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: run some experiments with random_search\n",
    "# Choose some value for trial. Train it on the data set. Test it on the test set. \n",
    "# Print the loss on the data set and the test set. Measure and print how long the training takes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85e169df",
   "metadata": {},
   "source": [
    "## P5: Bonus: Improvements (3 pts)\n",
    "Propose an improvement to the algorithms you have implemented for 4 and 5 and show that your improvements perform better than the original. Some examples of what you might try:\n",
    "* Choose values for $\\theta_0$ and $\\theta_1$ on a non-uniform grid\n",
    "* First find one of them, and fix it, and then refine on the other one\n",
    "* For random: sample according to a non-uniform distribution\n",
    "* First use a low resolution search to find the approximate values of  $\\theta_0$ and $\\theta_1$, then search for a more precise value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb552fcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: implement your improvements here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d1dd74e",
   "metadata": {},
   "source": [
    "TODO: Describe in one paragraph the conclusions you have drawn from your improvement experiments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5c0d4e8",
   "metadata": {},
   "source": [
    "## P6: Using the sklearn library (3 pts)\n",
    "\n",
    "Use ``sklearn.linear_model.LinearRegression`` to solve the same problem you previously solved using the grid search and random search.\n",
    "\n",
    "Compare the returned values with what you have achieved. Compare the parameters that had been found to the parameters you have found. Compare the speed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "136e4f63",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Implement here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1268d33c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Run performance experiments here. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b2dc65e",
   "metadata": {},
   "source": [
    "TODO: discuss the performance of the sklearn library implementation, compared to what you implemented."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39cba7e7",
   "metadata": {},
   "source": [
    "# Setup for the second part of the project\n",
    "For the questions P7-P10 we use linear regression on a multivariate setting. This time, there are 7 explanatory variables: ``sq_mt_built``, ``n_rooms``, ``n_bathrooms``, ``is_renewal_needed``, ``is_new_development`` and ``has_fitted_wardrobes``. \n",
    "\n",
    "We will first create the training and test data while doing some minimal data cleaning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f68a641",
   "metadata": {},
   "outputs": [],
   "source": [
    "# replacing the NA values with some sensible defaults\n",
    "# the way I was investigating these is by printing \n",
    "#    df[\"has_individual_heeating\"].value_counts(dropna=False) etc\n",
    "df_shuffled[\"has_individual_heating\"] = df_shuffled[\"has_individual_heating\"].fillna(False)\n",
    "df_shuffled[\"n_bathrooms\"] = df_shuffled[\"n_bathrooms\"].fillna(1)\n",
    "df_shuffled[\"has_individual_heating\"] = df_shuffled[\"has_individual_heating\"].fillna(False)\n",
    "df_shuffled[\"is_new_development\"] = df_shuffled[\"is_new_development\"].fillna(False)\n",
    "df_shuffled[\"has_fitted_wardrobes\"] = df_shuffled[\"has_fitted_wardrobes\"].fillna(False)\n",
    "\n",
    "xfields = [\"sq_mt_built\", \"n_rooms\", \"n_bathrooms\", \"has_individual_heating\", \\\n",
    "           \"is_renewal_needed\", \"is_new_development\", \"has_fitted_wardrobes\"]\n",
    "\n",
    "x = df_shuffled[xfields].to_numpy(dtype=np.float64)\n",
    "y = df_shuffled[\"buy_price\"].to_numpy(dtype=np.float64)\n",
    "training_data_x = x[:16000]\n",
    "training_data_y = y[:16000]\n",
    "test_data_x = x[16000:]\n",
    "test_data_y = y[16000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "848d0c5b",
   "metadata": {},
   "source": [
    "## P7: Implement grid search for multiple variables (3 pts)\n",
    "Implement the linear predictor model for multiple variables. Note that this time ``x`` will be an array of 7 values, and ``theta`` will be an array of 8 values. \n",
    "\n",
    "Then, implement a grid search function (similar to P3) but this time for the 7 explanatory variables. Pass the grids as an array into the grid variable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bde6d4ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: implement the predictor function here\n",
    "def predict_multi(x, theta):\n",
    "    y_hat = 0\n",
    "    return y_hat\n",
    "\n",
    "# TODO: implement the grid search function here \n",
    "def grid_search(training_data_x, training_data_y, grids):\n",
    "    theta = [0, 0]\n",
    "    return theta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "080d821a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: run experiments with your implementation for the grid search "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab609edb",
   "metadata": {},
   "source": [
    "TODO: describe here your experiences with implementing this problem, conclusions you draw. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "020b814a",
   "metadata": {},
   "source": [
    "## P8: Random search for multiple variables (3 pts)\n",
    "Implement the random search technique for the multiple variables. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37f3b832",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: implement the random seeach function here \n",
    "def grid_search(training_data_x, training_data_y):\n",
    "    theta = [0, 0]\n",
    "    return theta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c553a29d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: run experiments with your implementation for the"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba6f2531",
   "metadata": {},
   "source": [
    "TODO: describe here your experiments with the implementation for the random search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f8cd1c2",
   "metadata": {},
   "source": [
    "## P9: Use sklearn for linear regression in multiple variables (3 pts)\n",
    "Use ``sklearn.linear_model.LinearRegression`` to solve the same problem you previously solved using the grid search and random search.\n",
    "\n",
    "Compare the returned values with what you have achieved. Compare the parameters that had been found to the parameters you have found. Compare the speed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4f85619",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: implement here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b7fbdca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: run experiments here. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f3fb028",
   "metadata": {},
   "source": [
    "TODO: describe in one paragraph your experiences with implementing the multiple variable linear regression. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1c64ddf",
   "metadata": {},
   "source": [
    "## P10: Bonus: Data wrangling (3 pt)\n",
    "\n",
    "Perform data preprocessing / cleaning / wrangling on the multiple variable dataset. This might include changing the range, removing outliers, etc. The objective is to obtain a better performance by a regressor as measured on the test data. Document your experiments with plots etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4b909dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: insert the code you use to investigate the properties of the data here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d8585e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: implement the data transformations here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad84987c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: run experiments with the transformed data here. Measure the performance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5761a993",
   "metadata": {},
   "source": [
    "TODO: describe in one paragraph your experiences with the data wrangling process."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22ba6a81",
   "metadata": {},
   "source": [
    "## P11: Bonus: explore other linear regression techniques (3 pts)\n",
    "\n",
    "Explore the use of other models provided from the sklearn library for linear regression. \n",
    "\n",
    "https://scikit-learn.org/stable/modules/classes.html#module-sklearn.linear_model\n",
    "\n",
    "Try out two of them of your choice. Explain the results you obtained and compare them with other approaches. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e23bd502",
   "metadata": {},
   "source": [
    "# K-nearest neighbors"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16543248",
   "metadata": {},
   "source": [
    "## P12: K-nearest neighbors for the single variable case (3 pts)\n",
    "Implement the k-nearest neighbor algorithm for the single variable case. Given an x value, find the k closest values from the training data, and return their average. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "079009df",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: implement here \n",
    "def predict_k_nearest(x, k):\n",
    "    y_hat = 0\n",
    "    return y_hat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1bcbf2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: experiment here in terms of accuracy and speed. Experiment with multiple values of k"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de875fea",
   "metadata": {},
   "source": [
    "TODO: Write a paragraph about the results of the experiments. How does it compare \n",
    "to the other techniques you implemented above?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28028c18",
   "metadata": {},
   "source": [
    "## P13: K-nearest neighbors for the multiple variable case (3 pts)\n",
    "Implement the k-nearest neighbor algorithm for the multiple variable case. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fad0731b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: implement here \n",
    "def predict_k_nearest_multiple(x, k):\n",
    "    y_hat = 0\n",
    "    return y_hat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89d0a0f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: experiment here in terms of accuracy and speed. Experiment with multiple values of k. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ef574a1",
   "metadata": {},
   "source": [
    "TODO: Write a paragraph about the results of the experiments. How does it compare \n",
    "to the other techniques you implemented above?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d588b35b",
   "metadata": {},
   "source": [
    "## P14: Bonus: Experiment with the sklearn implementation of K-nearest neighbors (3 pts)\n",
    "Using the ``sklearn sklearn.neighbors.KNeighborsRegressor`` model, implement the multi-variable regression model. Run experiments with different values of the $k$ hyper-parameter. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9fb2179b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: run experiments here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43a0f077",
   "metadata": {},
   "source": [
    "TODO: Write a paragraph about the results of the experiments. How does it compare to the techniques you implemented for k-nearest neighbor in terms of accuracy and speed. Provide an explanation for the results. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}